Language identi cation of web documents using discrete HMMs

Automatic language identi cation in written text documents is an issue which deserves signi cant attention in the context of the ever-growing volume of web documents. This paper deals with language identi cation in the domain of electronic texts related to tourism. The proposed system is built on Hidden Markov Models (HMMs) that enable the modeling of character sequences. For this purpose, a parallel structure of ergodic discrete HMMs is used. During testing a previously unseen document is divided into its sentences and each of them is independently characterized in terms of the language it is written in. Experiments conducted on sentence-long documents demonstrated high identi cation rates.